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Objective 

Share  findings  from  a  project  that  explores  a  non- 
traditional  but  relevant  view  of  Six  Sigma. 
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Outline 

Scope  of  research  project 

Findings 

Path  forward 
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Software  &  IT  Best  Practices  and 
Six  Sigma:  Recent  History 

Many  papers  and  presentations  comparing  CMM(I)  &  Six  Sigma 

•  What  are  the  differences  and  the  similarities? 

•  How  do  they  compare  at  the  PA,  goal,  and  practice  level  of 
CMMI? 

•  How  do  they  compare  at  the  philosophy,  framework,  toolkit, 
and  metric  level  of  Six  Sigma? 

•  How  can  Six  Sigma  training  be  tailored  for  software? 

Some  papers  extending  to  ISO,  TSP,  Balanced  Scorecard, 
Measurement  &  Analysis  practices 

Technical  depth  and  reports  of  field  experience  have  increased 
with  time 

Venues  have  included  SEPG  Conferences,  STC  Conferences, 
Crosstalk,  and  ASQ’s  Software  Quality  Professional 
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Frequently  Asked  Questions 

How  do  I  leverage  Six  Sigma  with  the  SPI  initiatives  already 
underway  in  my  organization? 

Should  I  pick  Six  Sigma  or  CMMI?  How  do  I  convince  my 
management  that  it’s  not  an  either/or  decision? 

What  evidence  is  there  that  Six  Sigma  works  in  software  and 
systems  engineering? 

How  do  I  train  software  engineers  when  Six  Sigma  training  is 
geared  for  manufacturing? 

What  are  examples  of  Six  Sigma  projects  in  software?  IT? 
Isn’t  Six  Sigma  only  about  advanced  statistics? 

What  is  a  software  “opportunity”?  How  do  I  calculate  sigma? 
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Our  Observations 

Current  Reality 

•  A  small  number  of  organizations  are  excelling  in  their 
combined  usage  of  SEI  technologies  and  Six  Sigma  to 
speed  the  realization  of  bottom-line  benefits. 

•  The  DoD  is  not  getting  the  benefit  of  this. 

Desired  Future  Reality 

•  Defense  contractors,  DoD  organizations,  and 
commercial  organizations  achieve  bottom-line  impact 
faster  and  more  effectively  by  joining  SEI  technologies 
and  Six  Sigma. 

•  The  DoD  uses  Six  Sigma’s  strategic  aspects  to  select 
best  technology  solutions  to  achieve  its  mission. 
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Six  Sigma  as  Transition  Enabler 

The  SEI  is  conducting  a  research  project  to  explore  the 
feasibility  of  Six  Sigma  as  a  transition  enabler  for  software 
and  systems  engineering  best  practices. 

Hypothesis 

•  Six  Sigma,  used  in  combination  with  other  software, 
systems,  and  IT  improvement  practices,  results  in 

-  better  selections  of  improvement  practices  and 
projects 

-  accelerated  implementation  of  selected  improvements 

-  more  effective  implementation 

-  more  valid  measurements  of  results  and  success  from 
use  of  the  technology 

Achieving  Process  Improvement...  Better,  Faster,  Cheaper. 
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What  is  Transition? 

Technology  transition  is  the  process  of  creating  or  maturing 
a  technology,  introducing  it  to  its  intended  adopters,  and 
facilitating  its  acceptance  and  use,  where  technology  is 

•  Any  tool,  technique,  physical  equipment  or  method  of 
doing  or  making,  by  which  human  capability  is 
extended.” 

•  “The  means  or  capacity  to  perform  a  particular  activity.” 

Are  maturation,  introduction,  adoption,  implementation, 
dissemination,  rollout,  deployment,  or  fielding  part  of  your 
process  improvement  effort? 

Each  indicates  “transition  activities.” 

[Forrester],  [Schon],  [Gruber] 
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Research  Scope 

Primary  priorities 

•  CMMI  adoption 

•  IT  operations  and  security  best  practices 

-  Information  Technology  Infrastructure  Library  (ITIL) 

-  Control  Objectives  for  Information  and  Related  Technology 
(COBIT) 

Secondary  priority 

•  architecture  best  practices  and  Design  for  Six  Sigma 
Primary  audiences 

•  Software  Engineering  Process  Groups  (or  equivalent) 

•  Black  Belt  and  Green  Belt  Practitioners 

We  also  considered  the  relevance  of  this  project  for  technology 
developers. 
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A  Technology  Adoption  View 
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Project  Scope  X  Adoption  View 


SEI,  other  institutions 
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Our  Approach 

Research  Method:  Grounded  Theory 

Data  Collection 

•  Case  study  interviews 

•  Surveys 

•  Literature 

Data  Evaluation 

•  Qualitative  evaluation  of  text 

•  Conducted  by  Jeannine  and  Eileen  (no  non-SEI  parties) 

•  Findings  verification  via  research  participant  feedback 

“Feasibility”  Criterion 

•  Minimally,  a  hypothesis  required  one  credible  example 
of  its  application  to  be  deemed  “feasible.” 
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Collaborators  and  Contributors 

Collaborators  for  Research  Direction 

•  Lynn  Penn,  Lockheed  Martin  IS&S 

•  Bob  Stoddard,  Motorola 

•  Dave  Hallowell,  Six  Sigma  Advantage 

•  Gary  Gack,  Six  Sigma  Advantage 

•  John  Vu,  Boeing 

•  Lynn  Carter,  CMU  West 

•  Gene  Kim,  ITPI 

•  Kevin  Behr,  ITPI 

•  SEI  colleagues:  Julia  Allen,  Mike  Phillips,  Gian  Wemyss 

Other  contributors 

•  ISSSP 

•  isixsigma.com 

•  Case  study  and  survey  participants  (many  anonymous) 
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Outline 

Scope  of  research  project 

Findings 

Path  forward 
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Data  Breadth  and  Depth 

Results  are  based  on  the  project  data  set,  including  information 
from 

•  1 1  case  study  interviews 

•  8  partial  case  study  interviews 

•  survey  responses  from  more  than  80  respondents, 
representing  at  least  62  organizations  and  42  companies 

•  several  pilots  that  are  underway  to  try  new  ideas 

Because  of  the  proprietary  nature  of  our  data  and  the  non¬ 
disclosure  agreements  in  place,  the  results  in  this  public  briefing 
are  intentionally  at  a  high  level. 

Additional  detailed  reports  and  briefings  are  planned,  pending 
project  participant  approvals. 
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Context  of  Findings 

Participating  organizations  spanned 

•  Low-to-high  maturity 

•  Nearly  all  commercial  sectors 

•  Medium-to-large  in  size 

•  Organic,  contracted,  co-sourced  software  engineering, 
and  IT 

•  IT  development,  deployment,  and  operations 

Small  organizations’  and  DoD  organizations'  use  of  Six 
Sigma  in  this  context  has  neither  been  refuted  nor 
supported  by  project  evidence. 
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Context  of  Findings 

While  our  focus  was  on  CMMI,  ITIL,  and  COBIT,  we 
gathered  information  on  other  technologies  “in  play.” 

•  The  list  included  People  CMM  and  other  maturity 
models,  ATAM,  TSP,  ISO  Standards,  EIA  Standards 

The  Six  Sigma  adoption  decision 

•  was  frequently  made  at  the  enterprise  level,  with 
software,  systems,  and  IT  organizations  following  suit. 

•  was  driven  by  senior  management’s  previous 
experience  and/or  a  burning  business  platform. 

Six  Sigma  deployment  was  consistently  comprehensive. 
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Primary  Findings 

Six  Sigma  is  feasible  as  an  enabler  of  the  adoption  of 
software,  systems,  and  IT  improvement  models  and 
practices  (a.k.a.  “improvement  technologies”). 

The  CMMI  community  is  more  advanced  in  their  joint  use 
of  CMMI  &  Six  Sigma  than  originally  presumed. 


We  have  23  significant  (and  interrelated)  findings. 
A  selection  are  included  on  the  following  slides. 


©  2004  by  Carnegie  Mellon  University 


Version  1.0 


page  19 


CarnecpeMellon 

Software  Engineering  Institute 


General  Findings  1 

Six  Sigma  helps  integrate  multiple  improvement 
approaches  to  create  a  seamless,  single  solution. 

Rollouts  of  process  improvement  by  Six  Sigma  adopters 
are  mission-focused  as  well  as  flexible  and  adaptive  to 
changing  organizational  and  technical  situations. 

Six  Sigma  is  frequently  used  as  a  mechanism  to  help 
sustain  (and  sometimes  improve)  performance  in  the 
midst  of  reorganizations  and  organizational  acquisitions. 

Six  Sigma  adopters  have  a  high  comfort  level  with  a 
variety  of  measurement  and  analysis  methods. 
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General  Findings  2 

Six  Sigma  can  accelerate  the  transition  of  CMMI. 

•  Moving  from  CMMI  ML  3  to  5  in  9  months,  or  from  SW- 
CMM  Level  1  to  Level  5  in  3  years  (the  typical  move 
taking  12-18  months  per  level) 

•  Underlying  reasons  are  strategic  and  tactical 

When  Six  Sigma  is  used  in  an  enabling,  accelerating,  or 
integrating  capacity  for  improvement  technologies, 
adopters  report  quantitative  performance  benefits,  using 
measures  they  know  are  meaningful  for  their  organizations 
and  clients.  For  instance, 

•  ROI  of  3:1  and  higher,  reduced  security  risk,  and  better 
cost  containment 


[Hayes  95] 
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CMMI-Specific  Findings 

Six  Sigma  is  effectively  used  at  all  maturity  levels. 

Participants  assert  that  the  frameworks  and  toolkits  of  Six 
Sigma  exemplify  what  CMMI  high  maturity  requires. 

Case  study  organizations  do  not  explicitly  use  Six  Sigma 
to  drive  decisions  about  CMMI  representation,  domain, 
variant,  and  process-area  implementation  order;  however, 
participants  agree  that  this  is  possible  and  practical. 

CMMI-based  organizational  assets  enable  Six  Sigma 
project-based  learnings  to  be  shared  across  the  software 
and  systems  organizations,  and  thereby,  enable  a  more 
effective  institutionalization  of  Six  Sigma. 
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IT-Specific  Findings 

High  IT  performers  (development,  deployment,  and 
operations)  are  realizing  the  same  benefits  of  integrated 
process  solutions  and  measurable  results. 

•  However,  they  are  using  the  technologies  and  practices 
specific  to  their  domain  (ITIL,  COBIT,  sometimes 
CMMI). 

CMMI-specific  findings  apply  to  IT  organizations  who  have 
chosen  to  use  CMMI. 
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Architecture-Specific  Findings 

Multiple  organizations  are  pursuing  the  joint  use  of  Six 
Sigma,  CMMI,  and  ATAM,  focusing  on  the  strong 
connections  among  DFSS,  ATAM,  and  the  engineering 
process  areas  of  CMMI. 

Many  survey  respondents  are  in  organizations  currently 
implementing  both  CMMI  and  Six  Sigma  DMAIC  and 
many  are  in  organizations  progressing  or  DFSS. 

•  Of  those  implementing  DFSS,  the  majority  are  at  least 
progressing  with  CMMI  (but  some  are  not  using  CMMI 
at  all)  and  none  are  using  ATAM. 


There  is  much  untapped  potential  here! 
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Remaining  Hypotheses 

There  are  33  remaining  significant  inferences  and 
hypotheses. 

They  have  been  reviewed  by  case  study  participants. 

Most  of  the  reviewers  believe  that  most  (but  not  all)  of  the 
hypotheses  are  true;  however,  we  do  not  yet  have  case 
evidence. 
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Why  Does  It  Work  for  Transition? 

We  observe  that 

•  Six  Sigma  supports  more  effective  transition  because  it 

-  requires  alignment  with  business  drivers 

-  garners  effective  sponsorship 

-  supports  excellent  and  rational  decision  making 

-  aids  robust  implementation  or  change  management 

-  offers  credible  measures  of  results  for  investment 

•  The  latter  is  particularly  crucial  for  convincing  majority 
adopters  to  transition,  and  is  often  the  sticking  point  in 
failed  transitions  (popularly  labeled  after  Moore  as 
failing  to  “cross  the  chasm”). 


[Moore] 
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Outline 

Scope  of  research  project 

Findings 

Path  forward 
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Concluding  This  Project 

The  initial  report  on  this  project  is  available  on  the  SEI  website. 

Additional  publications  are  planned,  including 

•  detailed  reports 

•  how  SEI  and  other  technologies  add  value  to  Six  Sigma  only 
adopters’  organizations 

•  briefings  at  future  conferences 

•  internal  SEI  briefings 

Please  contact  Jeannine  or  Eileen  if  you 

•  would  like  to  be  notified  of  publications 

•  have  relevant  information  you  would  like  to  share 

•  can  apply  the  results  of  this  work 

•  have  input  on  our  proposed  path  forward  (next  slide) 
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Proposed  Paths  Forward 

Apply  specific  findings  to  further  development  work  on 
CMMI,  Product  Line  Practices,  and  Measurement  & 
Analysis  technologies. 

Characterize  robustness  of  Six  Sigma  as  a  transition 
enabler. 

•  Organizational  fit,  characteristics 

•  Technology  fit,  characteristics 

•  Measuring  transition  progress 

•  Extension  to  organizations  and  technologies  (individual 
and  combined)  not  yet  studied 

Use  component-based  development  methods  and  Six 
Sigma  methods  to  create  guidance  for  the  effective 
integration  and  deployment  of  multiple  models. 
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Contact  Information 

Jeannine  Siviy 

Senior  Member  of  the  Technical  Staff 
Software  Engineering  Institute 

Software  Engineering  Measurement  &  Analysis  Initiative 
Email:  jmsiviy@sei.cmu.edu 
Phone:  412.268.7994 

Eileen  Forrester 

Senior  Member  of  the  Technical  Staff 
Software  Engineering  Institute 
Email:  ecf@sei.cmu.edu 
Phone:  412.268.6377 
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